Multi-touch skins spanning three dimensions

ABSTRACT

One or more multi-touch skins can placed along three dimensions of an object. The one or more multi-touch skins enable multi-touch inputs during the operation of the object. The multi-touch inputs can be tracked to monitor the operation of the object and provide feedback to the operator of the object. The one or more multi-touch skins can further enable gestures for configuring and operating the object. The one or more multi-touch skins can also be used to implement any number of GUI interface objects and actions. A multi-touch skin that measures the force of a touch in one or more directions is also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention is a continuation-in-part of commonly assignedU.S. Application entitled “Gestures For Devices Having One Or More TouchSensitive Surfaces,” filed Jun. 13, 2007, which in turn claims thebenefit under 35 USC 119(e) of U.S. Provisional Application No.60/878,828, filed Jan. 5, 2007. The contents of the identified U.SApplication entitled “Gestures For Devices Having One Or More TouchSensitive Surfaces” and U.S. Provisional Application 60/878,828 areincorporated by reference herein.

FIELD OF INVENTION

This relates to a multi-touch skin placed along three-dimensions of anobject for enabling multi-touch inputs during the operation of theobject. This also relates to a multi-touch skin that measures the forceof a touch in one or more directions.

BACKGROUND OF THE INVENTION

Touch pads and touch screens are input devices for operating a device.With a touch pad, such as touch pads on a personal laptop computer, themovement of the input pointer on a display generally corresponds to therelative movements of the user's finger as the finger is moved along asurface of the touch pad. Touch screens are a type of display screenthat can include a touch-sensitive transparent panel that overlays thedisplay screen. When using a touch screen, a user typically makes aselection on the display screen by pointing directly to objects (such asGUI objects) displayed on the screen.

In both of these cases, the user moves one or more fingers on atwo-dimensional surface to operate the device. However, for theoperation of many devices, it is desirable, preferable or even necessaryfor the user to move his or her fingers over a three-dimensionalsurface. Thus, there is a need for enabling multi-touch inputs andgestures in the operation of such devices.

SUMMARY OF THE INVENTION

This is directed to a multi-touch skin placed along three dimensions ofan object. A single multi-touch skin can be configured to span all threedimensions or two or more multi-touch skins can be used to span thethree dimensions. The one or more multi-touch skins can be permanentlyattached to the object or temporarily placed on the object.

Once placed on the object, the one or more multi-touch skins enablemulti-touch inputs during the operation of the object. The multi-touchinputs can be tracked to monitor the operation of the object and providefeedback to the operator of the object.

The one or more multi-touch skins can further enable gestures forconfiguring and operating the object. The one or more multi-touch skinscan also be used to implement any number of GUI interface objects andactions.

This also relates to an improved multi-touch skin for measuring themagnitude (Z) of a touch. In one example, the multi-touch skin has acorrugated surface.

This also relates to a multi-touch skin that can measure not only themagnitude (Z) of a touch, but also the force of the touch in one or moredirections, i.e., the x, y and/or z directions. The four types of datacan enable a broad vocabulary of actions and gestures.

BRIEF DESCRIPTION OF THE DRAWINGS

This will be readily understood by the following description inconjunction with the accompanying drawings, wherein like referencenumerals designate like structural elements.

FIGS. 1A and 1B illustrate an exemplary embodiment in which onemulti-touch skin spans three dimensions of an object.

FIG. 2 illustrates a system of a multi-touch skin and a processingdevice according to an exemplary embodiment.

FIG. 3 is a flow diagram of multi-touch processing method in accordancewith an exemplary embodiment.

FIG. 4 is a flow diagram of a digital signal processing method inaccordance with one embodiment.

FIG. 5 illustrates an exemplary embodiment of two two-dimensionalmulti-touch skins spanning three dimensions of an object.

FIG. 6 is a multi-touch processing method in accordance with anexemplary embodiment of this invention.

FIGS. 7A and 7B illustrate an image in accordance with an exemplaryembodiment of this invention.

FIG. 8 illustrates a group of features in accordance with an exemplaryembodiment of this invention.

FIG. 9 is a parameter calculation method in accordance with an exemplaryembodiment of this invention.

FIG. 10 illustrates an exemplary embodiment of two multi-touch skinsspanning three dimensions of an object.

FIG. 11 illustrates an exemplary embodiment of a portable multi-touchskin.

FIG. 12 illustrates an exemplary embodiment of an improved multi-touchskin for measuring the magnitude (Z) of a touch.

FIGS. 13A and 13B illustrate an exemplary embodiment of a multi-touchskin that can measure the force of a touch in one or more directions.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following description of preferred embodiments, reference is madeto the accompanying drawings which form a part hereof, and in which itis shown by way of illustration specific embodiments in which theinvention can be practiced. It is to be understood that otherembodiments can be utilized and structural changes can be made withoutdeparting from the scope of the preferred embodiments of the invention.

FIGS. 1A and 1B illustrate an exemplary embodiment of a singlemulti-touch skin spanning three dimensions of an object. In accordancewith one embodiment, object 10 illustrated in FIGS. 1A and 1B can be apiece of sporting equipment, such as a tennis racket, golf club orhockey stick. Object 10 can include a handle 12 that is designed to begrasped by a player while playing the sport associated with object 10.FIG. 1A illustrates multi-touch skin 14 wrapping around the entirecircumference of handle 12 and extending along the length of handle 12.In this manner, multi-touch skin 14 spans three dimensions of object 10and enables multi-touch input as a player grasps handle 12 in theoperation of object 10.

The dimensions of multi-touch skin 14 will vary depending on the size ofobject 10 and how the player grasps the object. For example, thedimensions of multi-touch skin 14 for a dart will be much smaller than amulti-touch skin for a tennis racket, golf club or hockey stick, giventhe dart's size and how it is held by only two fingers. The shape ofmulti-touch skin 14 will also vary depending on how the player graspsobject 10. For example, the multi-touch skin 14 can be positioned in oneor more patches spanning three dimensions of an object if the object,such as a gun, is held with spread fingers.

Multi-touch skin 14 can be configured to recognize multiple touch eventsthat occur at different locations on surface 15 at the same time andmonitor the operation of the object 10. This can be achieved through amulti-touch sensing arrangement. In the exemplary arrangementillustrated in FIG. 1A, surface 15 of multi-touch skin 14 is dividedinto several independent and spatially distinct sensing nodes, such asnodes 16, 18, 20 and 22. The nodes are typically hidden from view andare dispersed about surface 15. Each node represents a differentposition on surface 15 and is capable of generating a signal at the sametime as other nodes. In the simplest case, a signal is produced eachtime an object is placed over a node. When, for example, handle 12 isgrasped by several fingers and a palm, multiple signals are generatedcorresponding to the nodes touched by the surface contact.

The number of nodes can be widely varied. The number of nodes generallydepends on the desired sensitivity as well as the transparency, ifdesired, of multi-touch skin 14. More nodes generally increasessensitivity, but reduces transparency (and vice versa). FIG. 1Aillustrates the nodes in a grid-like Cartesian system. However, as withthe number of nodes, the configuration of nodes can be widely varied.

In the embodiment of FIG. 1A, the nodes detect touches throughcapacitive sensing. Capacitive sensing nodes can be widely varied. Forexample, capacitive sensing nodes can be based on “self” capacitance or“mutual” capacitance.

In self capacitance, each node in the multi-touch skin can be providedby an individual charged electrode. As an object, such as a finger,approaches the surface of a multi-touch skin, the object couplescapacitively to the electrodes in close proximity to the object, therebystealing charge away from the electrodes. The amount of charge in eachof the electrodes is measured by a sensing circuit to determine theposition of the object.

In mutual capacitance, a multi-touch skin can include a two layer gridof spatially separated lines or wires. In the simplest case, the upperlayer can include lines in rows while the lower layer can include linesin columns (e.g., orthogonal). The nodes are provided at theintersections of the rows and columns. During operation, the rows arecharged and the charge couples capacitively to the columns at theintersection. As an object, such as a finger, approaches the surface ofthe multi-touch skin, the object couples capacitively to the rows at theintersections in close proximity to the object, thereby stealing chargeaway from the rows and therefore the columns as well. The amount ofcharge in each of the columns is measured by the sensing circuit todetermine the positions of multiple objects when they touch themulti-touch skin.

In either self-capacitance or mutual capacitance, each node worksindependently of the other nodes so as to produce simultaneouslyoccurring signals representative of different points on multi-touch skin14.

In the embodiment of FIG. 1A, multi-touch skin 14 employsself-capacitance and includes a plurality of capacitive sensingelectrodes as further illustrated in FIG. 1B. Each electrode correspondsto a node and, thus, represents a different coordinate on the surface ofmulti-touch skin 14. For example, electrode 30 in FIG. 1B corresponds tonode 16 in FIG. 1A, electrode 32 corresponds to node 18 and so on. Theelectrodes can be configured to receive capacitive input from one ormore objects, such as fingers, touching the surface 15 of multi-touchskin 14. When a finger is proximate to an electrode, the finger stealscharge as described above.

The electrodes can be connected to a capacitive sensing circuitincluding sensor ICs through traces that are positioned in the gapsfound between the spaced apart electrodes. The electrodes are spacedapart in order to electrically isolate them from each other as well asto provide a space for separately routing the traces. For example, inthe internal portion of multi-touch skin 14, illustrated in FIG. 1B,electrodes 30 and 32 are connected to sensor IC 34 via traces 36 and 38.Any number of sensor ICs may be used. For example, a single chip may beused for all electrodes, or multiple chips may be used for a single orgroup of electrodes. In FIG. 1B, sensor IC 34 is connected to a group ofelectrodes, including electrodes 30 and 32, along a row. While only twotraces are shown in FIG. 1B, it will be appreciated the number of tracesconnected to a given sensor IC can be more than shown if there are moreelectrodes in the row.

Each sensor IC can measure the capacitance at each electrode to which itis connected and report its findings or some form thereof to a hostcontroller. The sensor ICs may for example convert the analog capacitivesignals to digital data and thereafter transmit the digital data over aserial bus, such as bus 39, to a host controller or processor. In mostcases, the sensor ICs report tracking signals, which can be a functionof both the position of the electrode and the intensity of thecapacitance at the electrode. If the host controller or processor ispositioned on object 10, then bus 39 can be directly connected to thehost controller or processor for transmission of data. If the hostcontroller or processor is positioned apart from object 10, then thetransmission of data to the host controller or processor may be througha wired or wireless connection.

The electrodes, such as electrodes 30 and 32, and traces, such as traces36 and 38, can be made from any suitable conductive material. Dependingon the application, the material can be transparent as well. By way ofexample, the electrodes and traces may be formed from indium tin oxide(ITO). In addition, the sensor ICs, such as sensor IC 34, of the sensingcircuit can be electrically coupled to the traces using any suitabletechnique.

In one embodiment, the electrodes, traces and sensing circuit areattached to a flex circuit. The flex circuit can be a single-sidecircuit made of single copper conductor layer on a flexible dielectricfilm. Such single-sided flex circuits can be made as thin as 0.10 mm oreven smaller. The flex circuit can be fabricated with or without a coverlayer. The flex circuit can then be wrapped around and attached tohandle 12 as illustrated in FIGS. 1A and 1B.

Alternatively, one or more sensor ICs of the sensing circuit can beplaced directly on handle 12 such as on its bottom 13. In anotherimplementation, the electrodes, traces and sensing circuit can beembedded on handle 12 using any suitable patterning technique.

The distribution of the electrodes, such as electrodes 30 and 32, can bewidely varied. For example, the electrodes can be positioned almostanywhere in multi-touch skin 14. The electrodes can be positionedrandomly or in a particular pattern about multi-touch skin 14. For thelatter, the position of the electrodes 102 can depend on the coordinatesystem used. For example, the electrodes can be placed in an array ofrows and columns for Cartesian coordinates as illustrated in FIGS. 1Aand 1B or an array of concentric and radial segments for polarcoordinates. Within each array, the rows, columns, concentric or radialsegments can be stacked uniformly relative to the others or they can bestaggered or offset relative to the others. Additionally, within eachrow or column, or within each concentric or radial segment, theelectrodes can be staggered or offset relative to an adjacent electrode.

Furthermore, the electrodes, such as electrodes 30 and 32, can be formedfrom almost any shape whether simple (e.g., squares, circles, ovals,triangles, rectangles, polygons, and the like) or complex (e.g., randomshapes). Further still, the shape of the electrodes can have identicalshapes or they may have different shapes. For example, one set ofelectrodes can have a first shape while a second set of electrodes canhave a second shape that is different than the first shape. The shapesare generally chosen to maximize the sensing area and, if themulti-touch skin is transparent, to minimize optical differences betweenthe gaps and the transparent electrodes.

In addition, the size of the electrodes, such as electrodes 30 and 32,can vary according to the specific needs of each object. In some cases,the size of the electrodes can correspond to about the size of a fingertip. For example, the size of the electrodes can be on the order of 4-5mm². In other cases, the size of the electrodes can be smaller than thesize of the finger tip so as to improve resolution of the multi-touchskin (the finger can influence two or more electrodes at any one timethereby enabling interpolation). Like the shapes, the size of theelectrodes can be identical or they can be different. For example, oneset of electrodes can be larger than another set of electrodes.Moreover, any number of electrodes can be used. The number of electrodescan be determined by the size of the multi-touch skin as well as thesize of each electrode. In most cases, it would be desirable to increasethe number of electrodes so as to provide higher resolution, i.e., moreinformation can be used for such things as acceleration.

Although the sense traces, such as traces 36 and 38, can be routed avariety of ways, they can be routed in manner that reduces the distancethey have to travel between their electrode and the sensor circuit, andthat reduces the size of the gaps found between adjacent electrodes. Thewidth of the sense traces can also be widely varied. The widths can begenerally determined by the amount of charge being distributed therethrough, the number of adjacent traces, and the size of the gap throughwhich they travel. It is generally desirable to maximize the widths ofadjacent traces in order to maximize the coverage inside the gapsthereby creating a more uniform optical appearance if the skin istransparent.

In the illustrated embodiment, the electrodes, such as electrodes 30 and32, are positioned in a pixilated array. The electrodes are positionedin rows that extend to and from the sides of the multi-touch skin.Within each row,.the identical electrodes are spaced apart andpositioned laterally relative to one another (e.g., juxtaposed).

The sense traces for each row can be routed in two different directions.The sense traces on one side of the row can be routed to a sensor IC,such as sensor IC 34, located on the left side and the sense traces onthe other side of the row can be routed to another sensor IC located onthe right side of the multi-touch skin 14 (not shown). This is done tominimize the gap formed between rows. The gap can for example be held toabout 20 microns. As should be appreciated, the spaces between thetraces can stack thereby creating a large gap between electrodes. Ifrouted to one side, the size of the space would be substantially doubledthereby reducing the resolution of the multi-touch skin. Moreover, theshape of the electrodes, such as electrodes 30 and 32, can be in theform of a parallelogram, and more particularly a parallelogram withsloping sides.

Multi-touch skin 14 illustrated in FIG. 1B utilizes the concept ofself-capacitance as described above. It can be appreciated that othermulti-touch sensor detection methods including without limitation mutualcapacitance can be utilized. Multi-touch sensor detection methods aredescribed in commonly assigned co-pending applications, including U.S.application Ser. No. 10/840,862, published on May 11, 2006 as U.S.Patent Publication No. US2006/0097991, U.S. application Ser. No.11/428,522, published on Oct. 26, 2006 as U.S. Patent Publication No.US2006/0238522, and U.S. application Ser. No. 11/649,998 entitled“Proximity and Multi-Touch Sensor Detection and Demodulation,” filed onJan. 3, 2007, the entirety of each of which are hereby incorporatedherein by reference.

Once multi-touch skin 14 recognizes a multi-touch event on surface 15,it can produce tracking signals corresponding to the multiple touches.Multi-touch skin 14 can process the tracking signals itself or outputthe tracking signals to a device for processing. FIG. 2 is a blockdiagram of system 50 in accordance with one embodiment. In thisembodiment, multi-touch skin 52, positioned on an object such as a pieceof sporting equipment, outputs tracking signals to device 51.Specifically, multi-touch skin 52 is configured to receive input from aplayer's touch as he or she grasps the object's handle.

In most cases, multi-touch skin 52 can recognize touches 53 and theposition and magnitude of touches on its surface. Multi-touch skin 52reports touches 53 to device 51. Device 51 can be positioned on theobject or separated from the object. If device 51 is positioned on theobject, multi-touch skin 52 can report touches 53 to processor 56 towhich it is operatively coupled. If device 51 is separated from theobject, multi-touch skin 52 can report touches 53 to processor 56through, for example, a cable connection or a wireless connection.

Processor 56 is configured to execute instructions and to carry outoperations associated with device 51. For example, using instructionsretrieved from memory 58, processor 56 can control the reception andmanipulation of input and output data between components of device 51.Processor 56 can be a single-chip processor or can be implemented withmultiple components.

In most cases, processor 56 together with an operating system operate toexecute computer code and produce and use data. The computer code anddata may reside within a program storage block 58 that is operativelycoupled to processor 56. Program storage block 56 generally provides aplace to hold data that is being used by system 50. By way of example,program storage block 58 can include Read-Only Memory (ROM),Random-Access Memory (RAM), hard disk drive and/or the like. Thecomputer code and data could also reside on a removable storage mediumand loaded or installed onto the computer system when needed.

Device 51 can also include an input/output (I/O) device 54 that isoperatively coupled to processor 56. I/O device 54 may be integratedwith processor 56 or it may be a separate component as shown. I/O device54 is generally configured o control interactions with one or more I/Odevices. The I/O device 54 generally operates by exchanging data betweenprocessor 56 and external I/O devices that desire to communicate withthe processor. The external I/O devices and I/O device 54 typicallycommunicate through a data link. The data link may be a one way link ortwo way link. In some cases, the external I/O devices may be connectedto I/O device 54 through wired connections. In other cases, the externalI/O devices may be connected through wireless connections. By way ofexample, the data link may correspond to PS/2, USB, Firewire, IR, RF,Bluetooth or the like.

Device 51 can also include a display 60 that is operatively coupled toprocessor 56. Display 60 may be a separate component (peripheral device)or it may be integrated with the processor and program storage to form acomputer (all in one machine) on the object. Display 60 can beconfigured to display a graphical user interface (GUI) including perhapsa pointer or cursor as well as other information to the user or player.For example, display 60 may be a monochrome display, color graphicsadapter (CGA) display, enhance graphics adapter (EGA) display,variable-graphics-array (VGA) display, super VGA display, liquid crystaldisplay (e.g., active matrix, passive matrix and the like), cathode raytube (CRT), plasma displays and the like.

FIG. 3 is a flow diagram in accordance with one embodiment of thepresent invention that sets forth how a multi-touch event isinterpreted. The method generally begins at block 80 where a pluralityof sensing points on a multi-touch skin can be driven. For example, avoltage can be applied to the electrodes in a multi-touch skin, such asskin 14 of FIGS. 1A and 1B. Following block 80, the process flowproceeds to block 82 where outputs (voltage) from all the sensing pointscan be read. This block may include multiplexing and digitizing outputs.Following block 82, the process flow proceeds to block 84 where an imageor other form of data (signal or signals) of the multi-touch skin at onemoment in time can be produced and thereafter analyzed to determinewhere the objects, such as fingers, are touching the multi-touch skin.For example, as discussed below with respect to FIG. 4, the boundariesfor each unique touch can be calculated, and thereafter the coordinatesthereof can be found.

Once the coordinates of the touch are found, they can be analyzed asindicated in block 84. For example, the coordinates can be analyzed tomonitor how the object is being operated and provide feedback. In thecase of a tennis racket or a golf club, for example, the coordinates ofthe player's grasp of the handle can be detected. The coordinates canthen be analyzed to determine if the player has correctly grasped thehandle for a forehand stroke or a drive. The analysis may includeobtaining other information, such as the orientation of the racket faceor club head in relation to the grasp, to determine whether the playerhas correctly positioned his hand or hands on the handle.

Based on the analysis, an action can be performed. For example, adisplay, such as display 60 in FIG. 2, can be used to notify the playerthat he or she has correctly grasped the tennis racket or golf club forthe stroke to be performed. The display, for example, can display agreen light for a proper grasp and a red light for an incorrect grasp.If the grasp is incorrectly positioned on the handle, the display canfurther direct the player to correct the grasp. For example, the displaycan direct the player to move his or her index finger or thumb in agiven direction.

Following block 84, the process flow process to block 86 where thecurrent image or signal can be compared to a past image or signal inorder to determine a change in pressure, location, direction, speed andacceleration for each object on the multi-touch skin. This informationcan be subsequently used to perform an action as indicated in block 88.For example, if the grasp was incorrectly positioned on the handle, thecurrent image can be compared to the past image to determine whether theplayer correctly followed the display's instruction to move an indexfinger or thumb in a given direction. As another example, the currentimage can be compared to the past image to determine whether the playeris correctly positioning his or her grasp during a stroke or drive. Thedisplay can notify the player if his or her grasp is positionedcorrectly during the stroke or drive and provide direction.

FIG. 4 is a flow diagram of a digital signal processing method, inaccordance with one embodiment of the present invention. By way ofexample, the method can generally correspond to block 84 shown anddescribed in FIG. 3. The method generally begins at block 92 where theraw data is received. The raw data can be in a digitized form, and caninclude values for each node of the multi-touch skin. The values can bebetween 0 and 256 where 0 equates to the highest capacitive coupling (notouch pressure) and 256 equates to the least capacitive coupling (fulltouch pressure).

Following block 92, the process flow proceeds to block 94 where the rawdata is filtered. As should be appreciated, the raw data can includesome noise. The filtering process can be configured to reduce the noise.By way of example, a noise algorithm can be run that removes points thatare not connected to other points. Single or unconnected pointsgenerally indicate noise while multiple connected points generallyindicate one or more touch regions, which are regions of the multi-touchskin that are touched by objects such as fingers.

Following block 94, the process flow proceeds to block 96 where gradientdata is generated. The gradient data can indicate the topology of eachgroup of connected points. The topology can be based on the capacitivevalues for each point. Points with the lowest values are steep whilepoints with the highest values are shallow. As should be appreciated,steep points can indicate touch points that occurred with greaterpressure while shallow points can indicate touch points that occurredwith lower pressure.

Following block 96, the process flow proceeds to block 98 where theboundaries for touch regions are calculated based on the gradient data.In general, a determination can be made as to which points are groupedtogether to form each touch region.

In one embodiment, the boundaries can be determined using a watershedalgorithm. Generally speaking, the algorithm performs imagesegmentation, which is the partitioning of an image into distinctregions as for example the touch regions of multiple objects in contactwith the multi-touch skin. The concept of watershed initially comes fromthe area of geography and more particularly topography where a drop ofwater falling on a relief follows a descending path and eventuallyreaches a minimum, and where the watersheds are the divide lines of thedomains of attracting drops of water. Herein, the watershed linesrepresent the location of pixels, which best separate different objectstouching the multi-touch skin. Watershed algorithms can be widelyvaried. In one particular implementation, the watershed algorithmincludes forming paths from low points to a peak (based on the magnitudeof each point), classifying the peak as an ID label for a particulartouch region, associating each point (pixel) on the path with the peak.These steps are performed over the entire image map thus carving out thetouch regions associated with each object in contact with themulti-touch skin.

Following block 98, the process flow proceeds to block 100 where thecoordinates for each of the touch regions are calculated. This can beaccomplished by performing a centroid calculation with the raw dataassociated with each touch region. For example, once the touch regionsare determined, the raw data associated therewith can be used tocalculate the centroid of each touch region. The centroid may indicatethe central coordinates (x and y) of a given touch region. Thesecoordinates may be used to perform multi-touch tracking as indicated inblock 102. For example, the coordinates for each of the touch regionsmay be compared with previous coordinates of touch regions to determinepositioning changes of the objects touching the multi-touch skin orwhether or not touching objects have been added or subtracted or whethera particular object is being tapped. Digital signal processing methodsare described in commonly assigned co-pending applications, includingU.S. application Ser. No. 10/840,862, published on May 11, 2006 as U.S.Patent Publication No. US2006/0097991, which is hereby incorporatedagain by reference.

It can be appreciated that a multi-touch skin that spans threedimensions can be employed on any object where, for its operation,touching of the object in three dimensions is desired, preferred orrequired. Such a multi-touch skin can be applicable, without limitation,to video game input devices, toys, musical instruments, computingdevices and peripheral, medical devices, clothes and consumerelectronics devices.

It can also be appreciated that two or more two-dimensional multi-touchskins spanning three dimensions of an object can be used in place of asingle multi-touch skin spanning three dimensions of the object. FIG. 5illustrates an exemplary embodiment. Remote control 200 of FIG. 5remotely operates another device. The device can be, for example, aconsumer electronics product such as a television. Remote control 200can have a power button 210 and additional buttons 214, such as anumeric pad, for its operations.

A user can operate remote control 200 by holding it. For example, if theuser is right-handed, remote control 200 can be held such that theuser's thumb is positioned over the front side 225 of remote control200, the user's palm supports the back side 220 and his fingers curlaround the back side 220 and touch the left side 230 of remote control200.

In view of the anticipated placement of the user's thumb and fingers,remote control 200 has two two-dimensional multi-touch skins. A firstmulti-touch skin 240 is placed on the front side 225 of remote control200 in the x-y plane. A second multi-touch skin 245 is placed on theleft-side 230 of remote control 200 in the z-y. (A third multi-touchskin can be placed on the right-side 235 of remote control 200 forleft-handed users.) The user's thumb can touch multi-touch skin 240simultaneously with one or more of his fingers touching multi-touch skin245. In this manner, multi-touch skins 240 and 245 enable multi-touchinput as the user operates remote control 200.

Once each multi-touch skin recognizes a touch event on its respectivesurface, it can produce one or more tracking signals. The signals can beoutput to a device, such as illustrated in FIG. 2, for processing.

It should be noted multi-touch skins 240 and 245 are not contiguous.Moreover, as noted above, each multi-touch skin is only two dimensional.The placement of multi-touch skin 240 in the x-y plane and multi-touchskin 245 in the z-y plane nevertheless produces a combined output ofmulti-touch tracking signals as if a single multi-touch skin spans threedimensions of remote control 200. Thus, FIG. 5 illustrates an embodimentin which two two-dimensional skins are positioned along three dimensionsof an object to enable multi-touch inputs in the operation of theobject.

The placement of one or more multi-touch skins spanning three dimensionsof an object, such as remote control 200, enables gestures for operatingthe object. For example, the movement of the user's thumb acrossmulti-touch skin 240 can be used to control the selection of a channel.A downward movement of a thumb can numerically decrease the channelnumber, while an upward movement of the thumb can numerically increasethe channel number. Similarly, the movement of one or more fingers alongmulti-touch skin 24 can be used to reduce or increase volume. Thus, auser can grasp remote control 200 and simultaneously select the channeland volume through multi-touch gestures on skins 240 and 245. Of course,these two gestures can be also be performed separate in time. Whetherperformed simultaneously or not, these gestures can be tracked asmulti-touch inputs in the manner described above with respect to FIGS. 3and 4.

Gestures can also be used to configure the operation of the object oreven a given multi-touch skin. For example, the placement of anadditional thumb on multi-touch skin 240 can be used to configure remotecontrol 200 to control a different device, such as a DVD player. Theplacement of additional finger on multi-touch skin 240 can change themode for multi-touch skin 245 from volume selection to contrastselection. The movement of one or more fingers along 245 would thenreduce or increase the contrast as desired.

It should be appreciated that vocabulary of gestures is not limited tothe gestures described above. Additional examples of gestural strokesthat can be used as inputs for effecting interface commands, includinginteractions with UI elements are shown and described in commonlyassigned co-pending U.S. application Ser. No. 11/038,590, published asU.S. patent publication no. US2006/0026535, the entirety of which ishereby incorporated by reference, and commonly assigned co-pendingapplication Ser. No. 10/903,964, published as U.S. patent publicationno. US2006/0026521, the entirety of which is also hereby incorporated byreference and the parent application filed on Jun. 13, 2007.

It should also be appreciated that it is not necessary to always use ahuman finger to effect gestural input. Where possible, it can be alsosufficient to use a pointing device, such as a stylus, to effectgestural input.

It should also be appreciated that the one or multi-touch skins spanningthree dimensions of an object enable more than gestures. The one or moreskins can be used to implement any number of GUI interface objects andactions. FIG. 6 illustrates a multi-touch processing method 300, inaccordance with one embodiment of the invention, that allows amulti-touch skin having a touch screen to enable gestures and GUIobjects. The multi-touch processing method 300 can for example beperformed in the system shown in FIGS. 2 or 5, with multi-touch skin 245being a multi-touch touch screen.

The multi-touch processing method 300 generally begins at block 302where images can be read from a multi-touch input device, and moreparticularly a multi-touch touch screen, such as skin 245, illustratedin FIG. 5. Although the term “image” is used it should be noted that thedata can come in other forms. In most cases, the image read frommulti-touch skin 245 can provide magnitude (Z) as a function of position(x and y) for each sensing point or pixel of the skin. The magnitudecan, for example, reflect the capacitance measured at each point.

Following block 302, multi-touch processing method 300 proceeds to block304 where the image can be converted into a collection or list offeatures. Each feature can represent a distinct input such as a touch.In most cases, each feature can include its own unique identifier (ID),x coordinate, y coordinate, Z magnitude, angle Θ, area A, and the like.By way of example, FIGS. 7A and 7B illustrate a particular image 320 intime. In image 320, there are two features 322A and 322B based on twodistinct touches. The touches can for example be formed from a pair offingers touching multi-touch skin 245. As shown, each feature 322A and322B can include unique identifier (ID), x coordinate, y coordinate, Zmagnitude, angle Θ, and area A. More particularly, the first feature322A can be represented by ID₁, X₁, Y₁, Z₁, Θ₁, A₁ and the secondfeature 322B can be represented by ID₂, X₂, Y₂, Z₂, Θ₂, A₂. This datacan be outputted for example using a multi-touch protocol.

The conversion from data or images to features can be accomplished usingmethods described in copending U.S. patent application Ser. No.10/840,862, published as U.S. Patent Publication No. US2006/009771 anddescribed above with respect to FIG. 4. As disclosed therein, the rawdata can be typically received in a digitized form, and can includevalues for each node of the touch screen. The values can be between 0and 256 where 0 equates to no touch pressure and 256 equates to fulltouch pressure. Thereafter, the raw data can be filtered to reducenoise. Once filtered, gradient data, which indicates the topology ofeach group of connected points, can be generated. Thereafter, theboundaries for touch regions can be calculated based on the gradientdata (i.e., a determination can be made as to which points are groupedtogether to form each touch region). By way of example, a watershedalgorithm can be used as described above. Once the boundaries aredetermined, the data for each of the touch regions can be calculated(e.g., X, Y, Z, Θ, A).

Following block 304, multi-touch processing method 300 proceeds to block306 where feature classification and groupings can be performed. Duringclassification, the identity of each of the features can be determined.For example, the features can be classified as a particular finger,thumb, palm or other object. Once classified, the features can begrouped. The manner in which the groups can be formed can be widelyvaried. In most cases, the features can be grouped based on somecriteria (e.g., they carry a similar attribute). For example, the twofeatures shown in FIGS. 7A and 7B can be grouped together because eachof these features is located in proximity to each other or because theyare from the same hand. The grouping can include some level of filteringto filter out features that are not part of the touch event. Infiltering, one or more features can be rejected because they either meetsome predefined criteria or because they do not meet some criteria.

Following block 306, the multi-touch processing method 300 proceeds toblock 308 where key parameters for the feature groups can be calculated.The key parameters can include distance between features, x/y centroidof all features, feature rotation, total pressure of the group (e.g.,pressure at centroid), and the like. As shown in FIG. 8, the calculationcan include finding the centroid C, drawing a virtual line 330 to eachfeature from the centroid C, defining the distance D for each virtualline (D₁ and D₂), and then averaging the distances D₁ and D₂. Once theparameters are calculated, the parameter values can be reported. Theparameter values can be typically reported with a group identifier (GID)and number of features within each group (in this case three). In mostcases, both initial and current parameter values can be reported. Theinitial parameter values can be based on touch down, i.e., when the usersets their fingers on the touch screen, and the current values can bebased on any point within a stroke occurring after touch down.

As should be appreciated, blocks 302-308 can be repetitively performedduring a user stroke thereby generating a plurality of sequentiallyconfigured signals. The initial and current parameters can be comparedin later steps to perform actions in the system.

Following block 308, the process flow proceeds to block 310 where thegroup is or can be associated with a user interface (UI) element. UIelements can be buttons boxes, lists, sliders, wheels, knobs, pictures,documents, icons, etc. Each UI element can represent a component orcontrol of the user interface. In the case of remote control 200, the UIelement can be a channel selection button as an example. The applicationbehind the UI element(s) can have access to the parameter datacalculated in block 308. In one implementation, the application can rankthe relevance of the touch data to the UI element corresponding thereto. The ranking can be based on some predetermined criteria. The rankingcan include producing a figure of merit, and whichever UI element hasthe highest figure of merit, giving it sole access to the group. Therecan even be some degree of hysteresis as well (once one of the UIelements claims control of that group, the group sticks with the UIelement until another UI element has a much higher ranking). By way ofexample, the ranking can include determining proximity of the centroid(or features) to the image object associated with the UI element.

Following block 310, the multi-touch processing method 300 proceeds toblocks 312 and 314. The blocks 312 and 314 can be performedapproximately at the same time. From the user perspective, in oneembodiment, the blocks 312 and 314 appear to be performed concurrently.In block 312, one or more actions can be performed based on differencesbetween initial and current parameter values, and can also be based to aUI element to which they can be associated, if any. In block 314, userfeedback pertaining to the one or more actions being performed can beprovided. By way of example, user feedback can include display, audio,tactile feedback and/or the like.

FIG. 9 is a parameter calculation method 350, in accordance with oneembodiment of the invention. The parameter calculation method 350 can,for example, correspond to block 308 shown in FIG. 6. The parametercalculation method 350 generally begins at block 352 where a group offeatures can be received. Following block 352, the parameter calculationmethod 350 proceeds to block 354 where a determination can be made as towhether or not the number of features in the group of features haschanged. For example, the number of features can have changed due to theuser picking up or placing an additional finger. Different fingers canbe needed to perform different controls (e.g., tracking, gesturing). Ifthe number of features has changed, the parameter calculation method 350proceeds to block 356 where the initial parameter values can becalculated. If the number stays the same, the parameter calculationmethod 350 proceeds to block 358 where the current parameter values canbe calculated. Thereafter, the parameter calculation method 350 proceedsto block 360 where the initial and current parameter values can bereported. By way of example, the initial parameter values can containthe average initial distance between points (or Distance (AVG) initial)and the current parameter values can contain the average currentdistance between points (or Distance (AVG) current). These can becompared in subsequent steps in order to control various aspects of acomputer system.

The above methods and techniques can be used to implement any number ofGUI interface objects and actions. For example, gestures can be createdto detect and effect a user command to resize a window, scroll adisplay, rotate an object, zoom in or out of a displayed view, delete orinsert text or other objects, etc. Gestures can also be used to invokeand manipulate virtual control interfaces, such as volume knobs,switches, sliders, keyboards, and other virtual interfaces that can becreated to facilitate human interaction with remote control 200 or aconsumer electronic item.

Although the above embodiment can be described using a virtual controlchannel selection button or knob, in another embodiment, the UI elementcan be a virtual scroll wheel. As an example, the virtual scroll wheelcan mimic an actual scroll wheel such as those described in U.S. PatentPublication Nos. US2003/0076303A1, US2003/0076301A1, andUS2003/0095096A1, all of which are herein incorporated by reference.

It also be appreciated that multi-touch skins 240 and 245 do not have tobe identical in type. The selection of the type of multi-touch skin willdepend on the application. For example, multi-touch skin 245 can be anopaque touch pad employing self-capacitance, while multi-touch skin 240can be a transparent multi-touch touch screen employing mutualcapacitance as described above and in commonly assigned co-pendingapplications, including U.S. application Ser. No. 10/840,862, publishedon May 11, 2006 as U.S. Patent Publication No. US2006/0097991.

FIG. 10 illustrates yet another exemplary embodiment to show the broadapplicability of the present invention. FIG. 10 illustrates a steeringwheel 400. Steering wheel 400 can be used for the operation of an actualvehicle or a virtual vehicle. Steering wheel 400 has two multi-touchskins 410 and 420. Multi-touch skin 410 is positioned in the right upperquadrant of steering wheel 400, while multi-touch skin 420 is positionedin the left upper quadrant of steering wheel 400. Each multi-touch skinis wrapped around the entire circumference of steering wheel 400 to spanthree dimensions of the steering wheel.

When a user operates steering wheel 400, he places his right hand onmulti-touch skin 410 and his left hand on multi-touch skin 420. Eachskin thus receives a multi-touch input, and each skin outputscorresponding data. The data from each skin can be combined and analyzedby a processor, such as illustrated in FIG. 2 above, to performmulti-touch tracking as described above with respect to FIGS. 3 and 4.

For example, once the coordinates of each multi-touch are determined,they can be analyzed to determine if the user has correctly graspedsteering wheel 400. Based on the analysis, an action can be performed,such as notifying the user that he has not correctly grasped steeringwheel 400 or that his hands need to grasp steering wheel 400 at “10o'clock” and “2 o'clock.” As described above with respect to FIGS. 3 and4, a current image and a past image can be compared to determine if theuser has properly followed the notification and provide feedback ifnecessary.

Multi-touch skins 410 and 420 can also be used to operate the vehiclerelating to steering wheel 400 via gestural inputs. For example, thetapping of an index finger on multi-touch skin 410 can be used toperform channel selection of a radio in the vehicle. Each tap cancorrespond to a selection of the next available channel. The tapping ofan index finger and pinky finger on multi-touch skin 420 can be used toperform volume selection of the radio. Each tap of an index finger can,for example, correspond to a step increase in volume, while a tap of apinky finger can correspond to a step decrease in volume. In this way,the user can simultaneously change channels with one hand and controlvolume with the other hand, without have to focus his attention on thevehicle's radio. Of course, these two gestures can be also be performedseparate in time.

Multi-touch skins 410 and 420 can also be used together to configure theoperation of the vehicle (e.g., its steering wheel) and another skin.For example, the tapping of two fingers on multi-touch skin 410 can beused to change the mode for multi-touch skin 420 from volume selectionfor the radio to wiper speed selection. The tapping of an index fingeron multi-touch skin 420 would then correspond to a step increase inwiper speed while the tapping of a pinky finer would then correspond toa step decrease in wiper speed.

In the embodiments of FIGS. 2, 5 and 10, the one or more multi-touchskins are permanently attached to objects. FIG. 11 illustrates aportable multi-touch skin 510 that can be positioned along threedimensions of an object 500.

Portable multi-touch skin 510 is a flex circuit to which electrodes,traces and sensing circuit are attached. The flex circuit can then betemporarily placed or stretched over object 500. Multi-touch skin 510can be operatively coupled to a device, such as device 51 in FIG. 2, inorder for the user to provide setting information relating to the objectto be wrapped. For example, the setting information can indicate thatthe object is a guitar neck or a globe. The setting information can alsoprovide information relating to the placement of multi-touch skin 510vis-à-vis object 500. For example, the setting information can indicatearound which frets skin 510 is wrapped or where longitude 0° in relationto an edge of skin 510.

Once the setting information is established, the user can inputmulti-touch events to operate the device as described above with respectto FIGS. 2-10. For example, the user can perform chords on the guitarfret in conjunction with an instructional software on device 51. Theplayer's chords on multi-touch skin 510 can be tracked and then analyzedby the instructional software to determine if the player is properlyperforming the chords. As an another example, the user can touch twocountries on the globe and obtain distance information between the twocountries from device 51. It should be noted that all the capabilitiesdescribed with respect to device 51 can be implemented directly withskin 510.

In performing multi-touch tracking on a multi-touch skin, whetherpermanent or temporary, an image at given point in time can be taken ofthe touch event. The image, as described above with respect to FIG. 6,can be converted into a list features for the touch event. Each featurecan represent a distinct input such as a touch. As described above withrespect to FIG. 4 at block 100 and FIG. 6 as block 304, each feature caninclude coordinates. For example, as illustrated in FIGS. 7A and 7B,each feature 332A and 332B has an x-coordinate, a y-coordinate, amagnitude Z and so on.

Magnitude Z can represent the capacitance value for a given point orpixel and can be calculated for a given touch. If the current magnitudeZ is compared to a past magnitude Z, it can provide an indication of thepressure being applied on the multi-touch skin by the finger touchingthe skin. For example, an increase in magnitude Z can reflect anincrease in the pressure being applied by the finger on the multi-touchskin.

If a finger touches the multi-touch skin firmly on its initial contact,any further pressure being applied by the finger may only be reflectedin a marginal increase in magnitude Z. In other words, the measuredmagnitude Z can exhibit a non-linear response to the pressure beingapplied to the multi-touch skin. This can make tracking magnitude Zdifficult. It can also make performing actions based on a change inmagnitude Z difficult.

FIG. 12 is a partial front elevation view, in cross section of animproved multi-touch skin, in accordance with one embodiment of thepresent invention, for measuring the magnitude (Z) of a touch. Themulti-touch skin of FIG. 12 is a transparent touch screen employingmutual capacitance on a display. Specifically, the illustratedarrangement can include an LCD display 610 and a touch screen positionedover the LCD display 610. The LCD display 610 may correspond to anyconventional LCD display known in the art. Although not shown, the LCDdisplay 610 typically can include various layers including a fluorescentpanel, polarizing filters, a layer of liquid crystal cells, a colorfilter and the like.

The touch screen can include a transparent sensing layer 630 that ispositioned over a first glass member 620. The sensing layer 630 caninclude a plurality of sensor lines 690 positioned in columns (extendingin and out of the page). The first glass member 620 may be a portion ofthe LCD display 610 or it may be a portion of the touch screen. Forexample, it may be the front glass of the LCD display 610 or it may bethe bottom glass of the touch screen. The sensing layer 630 can betypically disposed on the glass member 620 using suitable transparentconductive materials and patterning techniques. In some cases, it may benecessary to coat the sensing layer 630 with material of similarrefractive index to improve the visual appearance, i.e., make moreuniform.

The touch screen also can include a transparent driving layer 660 thatis positioned over a second glass member 650. The second glass member650 can be positioned over the first glass member 620. The sensing layer630 is therefore sandwiched between the first and second glass members620 and 650. The second glass member 650 can provide an insulating layerbetween the driving and sensing layers 630 and 660. The driving layer660 can include a plurality of driving lines 695 positioned in rows(extending to the right and left of the page). The driving lines 695 canbe configured to intersect or cross the sensing lines 690 positioned incolumns in order to form a plurality of capacitive coupling nodes. Likethe sensing layer 630, the driving layer 660 can be disposed on theglass member using suitable materials and patterning techniques.Furthermore, in some cases, it may be necessary to coat the drivinglayer 660 with material of similar refractive index to improve thevisual appearance. Although the sensing layer is typically patterned onthe first glass member, it should be noted that in some cases it may bealternatively or additionally patterned on the second glass member.

The touch screen also can include a cover sheet 690 disposed over thedriving layer 660. The driving layer 660 is therefore sandwiched betweenthe second glass member 650 and cover sheet 690. Cover sheet 690 canserve to protect the under layers.

The cover sheet 690 can be made from any suitable clear material, suchas hard plastic. As illustrated in FIG. 12, cover sheet 690 iscorrugated. Sections 685 of the cover sheet (extending in and out of thepage) are raised, thereby creating valleys 687 (also extending in andout the page). When a finger touches cover sheet 690, the surfacecontact of the finger steals charge away from the corresponding rows andcolumns of the touch screen as described above with respect to mutualcapacitance. The corrugated surface can limit the amount of charge beingstolen, because it limits the surface contact of the finger, i.e., frombeing placed flush on the surface of the touch screen. Thus, the initialvalue of magnitude Z for a given touch on a corrugated surface is lessthan it would be on a flush surface.

As the user presses more firmly, his finger is pressed against theraised and valley sections. This spreads the finger and allows thefinger to steal more charge, thereby producing an increase in the valueof magnitude Z. The increase will be reflected in view of the initial,lower magnitude Z, thereby providing a more linear response between thepressure of a user's touch and the measured magnitude Z.

It can also be appreciated that cover sheet 690 is not limited to acorrugated surface as illustrated in FIG. 12. Any textured surface thatlimits initial surface contact and requires additional force to increasesurface contact can be used. For example, a surface having an array ofdimples can be used.

It can also be appreciated that the cover sheet 690 is not limited to ahard material, such as plastic. It can be a surface that stretches orbends to the touch. For example, cover sheet 690 can be made of rubber.The elasticity provided by the soft rubber can be used to further makethe measured magnitude Z closely track the pressure of a user's touch.

It can also be appreciated that the cover sheet 690 is not limited to atransparent cover sheet as illustrated in FIG. 12. The corrugated coversheet can, for example, be opaque.

It can also be appreciated that magnitude Z as measured can be used toperform actions and enable gestures. As described above with respect toFIGS. 3, 4 and 6, magnitude Z is a coordinate of a touch that can betracked. For example, a continuously increasing magnitude Z by a thumbon multi-touch skin 240 in FIG. 5 can indicate that the user wishes toquickly change the channels. Remote control 200 can then perform a quickchannel change, while multi-touch skin 240 can display a button that ispressed down in one direction to indicate a quick channel change in thatdirection.

The touch screen can also include various bonding layers 640 and 670.The bonding layers 640 and 670 can bond the glass members 620 and 650 aswell as the corrugated cover sheet 690 together to form the laminatedstructure and to provide rigidity and stiffness to the laminatedstructure. In essence, the bonding layers 640 and 670 help to produce amonolithic sheet that is stronger than each of the individual layerstaken alone. In most cases, the first and second glass members 620 and650 as well as the second glass member and the protective sheet 650 and690 can be laminated together using a bonding agent such as glue. Thecompliant nature of the glue may be used to absorb geometric variationsso as to form a singular composite structure with an overall geometrythat is desirable. In some cases, the bonding agent includes an indexmatching material to improve the visual appearance of the touch screen.

With regards to configuration, each of the various layers may be formedwith various sizes, shapes, and the like. For example, each of thelayers may have the same thickness or a different thickness than theother layers in the structure. In the illustrated embodiment, the firstglass member 620 can have a thickness of about 1.1 mm, the second glassmember 650 can have a thickness of about 0.4 mm. The thickness of thebonding layers 640 and 670 typically varies in order to produce alaminated structure with a desired height. Furthermore, each of thelayers may be formed with various materials. By way of example, eachparticular type of layer may be formed from the same or differentmaterial. For example, any suitable glass or plastic material may beused for the glass members. In a similar manner, any suitable bondingagent may be used for the bonding layers 640 and 670.

FIGS. 13A and 13B illustrate an exemplary embodiment of a multi-touchskin that can measure not only the magnitude (Z) of a touch, but alsothe force of the touch in one or more directions, i.e., the x, y and/orz directions. The four types of data can enable a broad vocabulary ofgestures.

FIG. 13A is a partial front elevation view, in cross section, of amulti-touch skin 700. Multi-touch skin 700 can include a base member 700upon which a driving layer 720 is placed. As illustrated in FIG. 13B,driving layer 720 can include a plurality of driving lines, such asdriving line 721, positioned to extend from the right and left of thepage. Multi-touch skin 700 can also include an insulating member 730positioned between driving layer 720 and sensing layer 740. Sensinglayer 740 can include a plurality of sensor lines 745, 746 and so on asillustrated in FIGS. 13A and 13B. The distance between two adjacentsensor lines can be 2.5 mm.

As illustrated in FIG. 13B, the sensor lines can extend in columnsorthogonal to driving lines. The intersection of the sensor lines anddriving lines produce sensing nodes. FIG. 13B illustrates sensing nodesA, B, C, D and E corresponding to the intersection of sensing lines745-749 and driving line 721. Based on this arrangement of sensingnodes, multi-touch skin 700 can perform multi-touch tracking throughmutual capacitance as described above.

A cover member 750 can be positioned over sensing layer 750. Covermember 750 can be made of an elastic material, such as soft rubber.Cover member 750 can be opaque, partially transparent or fullytransparent depending on the application. Dispersed within cover member750 can be a plurality of slugs 760, 761 and so on. The slugs can bemade of metal and can be positioned between every other sensing line anddriving line as illustrated in FIGS. 13A and 13B.

When a user touches cover member 750, cover member 750 can becomedeformed. Its deformation can change the position of one or more slugschanges in relation to its adjacent sensing and driving lines. Forexample, in a simple case of a touch in only the x-direction and towardthe right of the page, slugs 760 and 761 will be moved to the right ofthe page. The slugs will be positioned closer to sensing lines 746 and748 (and nodes B and D) and farther from sensing lines 745 and 747 (andnodes A and C) than in their respective original positions. The changein position of the slugs 760 and 761 can affect the capacitance betweenthe applicable sensing lines and driving lines, between adjacent sensinglines and between adjacent driving lines.

The capacitance at each applicable node can be calculated to provide ameasure of the force in the x-direction. For example, the signalreflecting the capacitance values over time at nodes A-E can be analyzedto measure the force in the x-direction. For example, the values fornodes B and D can be added while the values relating to nodes A, C and Ecan be subtracted as a spatial filter to provide a measure of the forcein the x-direction. Other data, such as touch Z between two nodes byadding the signals at the two nodes.

It can be appreciated that the touch being applied to cover member 750can be in more than just the x-direction. However, the force in they-direction and the z-direction can be measured by applying the sameapproach to the nodes affected by the movement of the slugs in the y andz-direction.

Multi-touch skin 700 can accordingly be capable of providing four typesof image data, i.e., magnitude Z and force data relating to one or moredirections. Such data can be used to enable a broad vocabulary ofactions and gestures. For example, any device that any form of velocityor steering control in a device, such as joystick or steering wheel, canbe achieved with multi-touch skin 700. As another, example, with respectto the exemplary embodiment of remote control 200 in FIG. 5, a user maybe faced with selecting a program through a matrix of channels (in rows)and times (in columns) displayed on a television. The user can roll histhumb on multi-touch skin 240 to move along a row of the matrix, e.g., agiven channel. Multi-touch skin 240 can measure not only the directionof the roll, e.g., in the x-direction and to the right, but also theforce of the roll in the x-direction. Such data can be used to controlhow fast the user scrolls along the row of the matrix, with a greaterforce providing a faster scroll. It can be appreciated that vocabularyof action and gestures is not limited to such gestures or devices.

It can also be appreciated that the distribution of slugs may be widelyvaried. For example, if the sensing and driving lines are positioned inan array of concentric and radial segments, the slug 760 can also bedistributed in along concentric and radial segments. It can also beappreciated that the distribution of slugs can be random or based on aparticular pattern.

It can also be appreciated that the measure of the force in the x, yand/or z-directions can be achieved in ways other than as illustrated inFIGS. 13A and 13B. For example, rubber cilia as opposed to a rubbercover with slugs can be used to achieve differences in capacitancebetween adjacent rows, adjacent columns and overlapping rows and columnsas the cilia is pushed in a given direction.

Many alterations and modifications can be made by those having ordinaryskill in the art without departing from the spirit and scope of theinvention. Therefore, it must be understood that the illustratedembodiments have been set forth only for the purposes of example andthat they should not be taken as limiting the invention as defined bythe following claims. For instance, although many of the embodiments ofthe invention are described herein with respect to personal computingdevices, it should be understood that the present invention is notlimited to desktop or laptop computers, but is generally applicable toother computing applications such as mobile communication devices,standalone multimedia reproduction devices, etc.

The words used in this specification to describe the invention and itsvarious embodiments are to be understood not only in the sense of theircommonly defined meanings, but to include by special definition in thisspecification structure, material or acts beyond the scope of thecommonly defined meanings. Thus if an element can be understood in thecontext of this specification as including more than one meaning, thenits use in a claim must be understood as being generic to all possiblemeanings supported by the specification and by the word itself.

The definitions of the words or elements of the following claims are,therefore, defined in this specification to include not only thecombination of elements which are literally set forth, but allequivalent structure, material or acts for performing substantially thesame function in substantially the same way to obtain substantially thesame result. In this sense it is therefore contemplated that anequivalent substitution of two or more elements can be made for any oneof the elements in the claims below or that a single element can besubstituted for two or more elements in a claim.

Insubstantial changes from the claimed subject matter as viewed by aperson with ordinary skill in the art, now known or later devised, areexpressly contemplated as being equivalently within the scope of theclaims. Therefore, obvious substitutions now or later known to one withordinary skill in the art are defined to be within the scope of thedefined claim elements.

The claims are thus to be understood to include what is specificallyillustrated and described above, what is conceptually equivalent, andwhat can be obviously substituted. For instance, the term “computer” or“computer system” as recited in the claims shall be inclusive of atleast a desktop computer, a laptop computer, or any mobile computingdevice such as a mobile communication device (e.g., a cellular orWi-Fi/Skype phone, e-mail communication devices, personal digitalassistant devices), and multimedia reproduction devices (e.g., iPod, MP3players, or any digital graphics/photo reproducing devices).

1. A three-dimensional object comprising: a first surface along atwo-dimensional plane; a second surface along a two-dimensional planedifferent than the two-dimensional plane of the first surface; and atleast one touch sensitive skin placed on at least a portion of the firstsurface and the second surface.
 2. The three-dimensional object of claim1 wherein the at least one touch sensitive skin includes an array ofcapacitance sensing nodes.
 3. The three-dimensional object of claim 2wherein the at least one touch sensitive skin includes an electricallyisolated electrode at each capacitance sensing node, each electrodehaving an individual trace for operatively coupling to capacitivemonitoring circuitry.
 4. The three-dimensional object of claim 3 whereinsaid capacitive monitoring circuitry includes one or more integratedcircuits for monitoring the capacitance at each of the electrodes. 5.The three-dimensional object of claim 4 further comprising a processingdevice positioned on the touch sensitive skin, said processing deviceoperatively coupled to the capacitive monitoring circuitry.
 6. Thethree-dimensional object of claim 3 wherein the electrodes are placed inrows and columns.
 7. The three-dimensional object of claim 1, where afirst touch sensitive skin is placed on the first surface and a secondtouch sensitive skin is placed on the second surface.
 8. Thethree-dimensional object of claim 1 wherein the at least one touchsensitive skin is opaque.
 9. The three-dimensional object of claim 1wherein the at least one touch sensitive skin is permanently attached toat least a portion of the first surface and the second surface.
 10. Athree-dimensional object of claim 1 wherein the object is a consumerelectronics device.
 11. A three-dimensional object of claim 10 whereinthe consumer electronics device is a remote control, a television, acomputer, a mobile telephone or a digital media player.
 12. Athree-dimensional object of claim 1 wherein the object is an electronicmusical instrument.
 13. A three-dimensional object of claim 1 whereinthe object is a handle of a sporting device.
 14. A computer implementedmethod for performing operations on a hand-held device having a display,a first surface along a two-dimensional plane, a second surface along atwo-dimensional plane different than the two-dimensional plane of thefirst surface and at least one touch sensitive skin placed on at least aportion of the first surface and the second surface, said methodcomprising: detecting a touch input on the touch sensitive skin at eachof the first surface and the second surface; separately recognizing thetouch input on each surface; and reporting touch data based on therecognized touch inputs.
 15. The computer implemented method of claim 14further comprising: comparing touch data received at two differenttimings; and performing an action based on the comparison of touch data.16. The computer implemented method of claim 14 further comprising:identifying at least one gesture based on the reported touch data; andperforming an action based on the identified gesture for operating thedevice.
 17. The computer implemented method of claim 14 furthercomprising: generating a user interface element on the display based onthe reported touch data.
 18. The computer implemented method of claim 17wherein the display is part of the touch-sensitive skin.
 19. Thecomputer implemented method of claim 17 further comprising: detecting atouch input on the touch sensitive skin at the displayed user interfaceelement; and performing an action related to the user interface elementbased on the detected touch input.
 20. A touch sensitive skincomprising: a surface for detecting multiple touches that occur at thesame time and at distinct locations on the surface, said touch sensitiveskin configured to produce distinct signal representative of thelocation for each touch, wherein said surface is textured.
 21. The touchsensitive skin of claim 20 wherein the textured surface is corrugated.22. The touch sensitive skin of claim 20 wherein the textured surface isdimpled.
 23. The touch sensitive skin of claim 20 wherein the texturedsurface is made of rubber.
 24. The touch sensitive skin of claim 20wherein the textured surface is made of hard plastic.
 25. A touchsensitive skin comprising: a capacitive sensing medium, said sensingmedium comprising a first layer having a plurality of conductive linesthat are electrically isolated from one another; and a second layerspatially separated from the first layer and having a plurality ofconductive lines that are electrically isolated from one another, thesecond conductive lines being positioned transverse to the firstconductive lines and forming nodes at the intersection of transverselines, said nodes positioned at different locations on the touchsensitive skin; and a cover member positioned over said capacitivesensing medium, said cover member comprising a plurality of conductiveunits, each unit positioned between at least two nodes.
 26. The touchsensitive skin of claim 25 wherein each conductive unit is made ofmetal.
 27. The touch sensitive skin of claim 25 wherein the cover memberis made of rubber.
 28. The touch sensitive skin of claim 25 wherein eachunit is positioned between every other group of at least two nodes. 29.A computer implemented method for performing operations on a hand-helddevice having a display and a touch sensitive skin, said touch sensitiveskin comprising a capacitive sensing medium having a plurality of nodesat the intersection of transverse electrically isolated conductive linesand a cover member positioned over said capacitive sensing medium, saidcover member comprising a plurality of conductive units, each unitpositioned between at least two nodes, said method comprising:monitoring the capacitance at each node; detecting a touch input on thecover member by detecting a change in capacitance at one or more nodesbased on the movement of one or more conductive units from the touch;and determining a coordinate of the touch input based on the determinedchange in capacitance.
 30. The computer implemented method of claim 29wherein determining a coordinate of the touch input comprisesdetermining a force of the touch in one or more directions.
 31. Thecomputer implemented method of claim 30 further comprising: performingan action based on the determined force in one or more directions. 32.The computer implemented method of claim 30 further comprising:identifying at least one gesture based on the determined force in one ormore directions; and performing an action based on the identifiedgesture for operating the device.
 33. The computer implemented method ofclaim 30 further comprising: generating a user interface element on thedisplay based on the determined force in one or more directions.
 34. Thecomputer implemented method of claim 33 wherein the display is part ofthe touch sensitive skin.
 35. The computer implemented method of claim34 further comprising: detecting a touch input on the touch sensitiveskin at the displayed user interface element; and performing an actionrelated to the user interface element based on the detected touch input.